Structured Prediction with Output Embeddings for Semantic Image Annotation

Multi-utility Learning: Structured-Output Learning with Multiple Annotation-Specific Loss Functions

Structured-output learning is a challenging problem; particularly so because of the difficulty in obtaining large datasets of fully labelled instances for training. In this paper we try to overcome this difficulty by presenting a multi-utility learning framework for structured prediction that can learn from training instances with different forms of supervision. We propose a unified technique f...

Semantic-Based Image Retrial in the VQ Compressed Domain using Image Annotation Statistical Models

VSE++: Improved Visual-Semantic Embeddings

We present a new technique for learning visual-semantic embeddings for crossmodal retrieval. Inspired by the use of hard negatives in structured prediction, and ranking loss functions used in retrieval, we introduce a simple change to common loss functions used to learn multi-modal embeddings. That, combined with fine-tuning and the use of augmented data, yields significant gains in retrieval p...

Learning Structured Semantic Embeddings for Visual Recognition

Numerous embedding models have been recently explored to incorporate semantic knowledge into visual recognition. Existing methods typically focus on minimizing the distance between the corresponding images and texts in the embedding space but do not explicitly optimize the underlying structure. Our key observation is that modeling the pairwise image-image relationship improves the discriminatio...

Semantic hierarchies for image annotation: A survey

In this survey, we argue that using structured vocabularies is capital to the success of image annotation. We analyze literature on image annotation uses and user needs, and we stress the need for automatic annotation. We briefly expose the difficulties posed to machines for this task and how it relates to controlled vocabularies. We survey contributions in the field showing how structures are ...